Skip to main content

FeatureLayer queryFeatures Pagination and Dynamic Map Coloring

Scope: 2D network map v2 hydraulic snapshot coloring (applyV2SnapshotDynamicRenderers.js) and the 3D map’s featureId → engineIndex index build. Static SSOT class-breaks coloring does not use this pagination helper.

Table of Contents​

  1. Symptom: Grey Features on Large Networks
  2. Root Cause
  3. Fix Overview
  4. Pagination: queryAllFeaturesAttributesOnly
  5. Primary Path vs Fallback
  6. When Pagination Runs
  7. Where maxRecordCount Comes From
  8. File Reference

1. Symptom: Grey Features on Large Networks​

On the 2D network map, when a layer used dynamic v2 snapshot coloring (dynamic:EN_* legend modes) and the network had more features than a single ArcGIS query could return:

  • The first chunk of junctions/pipes were colored correctly (yellow→red for nodes, cyan→blue for links).
  • Remaining features appeared grey (the UniqueValueRenderer defaultSymbol).

This was most visible on models with thousands of assets and on the UniqueValue fallback path (see below). Smaller networks often looked fine because everything fit in one query page.

2. Root Cause​

Dynamic coloring on the fallback path builds a UniqueValueRenderer: one uniqueValueInfos entry per feature, keyed by the layer’s id field (e.g. FID / OBJECTID), with colors taken from the v2 snapshot joined via Node_Index / Link_Index.

To build that renderer, the code must query every feature on the hosted FeatureLayer and match each row to snapshot metrics.

Before the fix, the code used a single layer.queryFeatures({ where: '1=1', ... }) call. ArcGIS feature services enforce a per-request record limit (maxRecordCount on the service/layer—often 1000 from our publish pipeline or 2000 as a common hosted default).

Only the first page of features was returned. uniqueValueInfos was built from that subset. Every other feature had no matching value in the renderer and drew with defaultSymbol → grey.

  query #1 (num = maxRecordCount)  →  features 1..N     →  colored
  features N+1 .. total            →  not in renderer  →  grey

3. Fix Overview​

Two complementary changes address the problem at different scales:

ApproachPurpose
Arcade + SimpleRenderer (preferred)Avoid per-feature UniqueValueRenderer; color via engineIndex lookup in an Arcade Dictionary embedded in a color visual variable. No full-layer feature query needed for coloring.
Paginated queryFeatures (fallback)When the Arcade path cannot be used, loop queryFeatures with start / num until exceededTransferLimit is false, then build UniqueValueRenderer from all features.

Implementation lives in:

  • src/map_utils/esri/applyV2SnapshotDynamicRenderers.js
  • src/map_utils/esri/snapshotDynamicArcade.js

4. Pagination: queryAllFeaturesAttributesOnly​

async function queryAllFeaturesAttributesOnly(layer, outFields = ['*']) {
  await layer.when();
  const pageSize =
    layer.maxRecordCount ??
    layer.capabilities?.query?.maxRecordCount ??
    layer.layerInfo?.maxRecordCount ??
    2000;
  const all = [];
  let start = 0;
  for (;;) {
    const q = await layer.queryFeatures({
      where: '1=1',
      outFields,
      returnGeometry: false,
      start,
      num: pageSize
    });
    const batch = q.features || [];
    all.push(...batch);
    if (!q.exceededTransferLimit) break;
    start += batch.length;
  }
  return all;
}

Behavior​

  1. Wait for the layer to load (layer.when()).
  2. Resolve pageSize from the loaded layer (see §7).
  3. Repeatedly call queryFeatures with:
    • where: '1=1'
    • returnGeometry: false (attributes only—cheaper)
    • start — offset for the next page
    • num: pageSize — page size for this request
  4. Stop when exceededTransferLimit is false (no more pages).
  5. Advance start by batch.length (not blindly by pageSize) so partial last pages are handled correctly.

Callers​

CallerUse
applySnapshotDynamicNodeLayerUniqueValueFallback2D dynamic node coloring fallback
applySnapshotDynamicLinkLayerUniqueValueFallback2D dynamic link coloring fallback
buildIdToEngineIndexMapFrom2DLayer3D map: one-time featureId → engineIndex map from junction/pipe 2D layers

5. Primary Path vs Fallback​

Primary path (no pagination)​

When the hosted layer has Node_Index / Link_Index (or aliases resolved in snapshotDynamicArcade.js) and the serialized metric map fits under SNAPSHOT_ARCADE_DICTIONARY_MAX_CHARS (450 000), the renderer is a single SimpleRenderer with a color visual variable. The snapshot is embedded in Arcade; the client evaluates color per feature without listing every id in uniqueValueInfos.

Fallback path (pagination required)​

Pagination runs only when:

  • The layer is missing engine-index fields, or
  • The Arcade payload would be too large for the client.

Then applySnapshotDynamic*LayerUniqueValueFallback calls queryAllFeaturesAttributesOnly and builds UniqueValueRenderer from the full feature set.

6. When Pagination Runs​

Pagination is runtime behavior, not app-startup configuration.

2D network map (EsriNetworkMapPage.jsx)​

Triggered by useEffect hooks when:

  • snapshotMatchesSlider && scenarioSnapshot are available, and
  • The layer legend mode is a dynamic SSOT metric (dynamic:EN_*), and
  • The code path falls through to the UniqueValue fallback (see §5).

Typical retriggers: hydraulic slider timestep change, scenario results load, legend mode change, layer reload.

The primary Arcade path does not call queryAllFeaturesAttributesOnly.

3D network map (Esri3DNetworkMapPage.jsx)​

buildIdToEngineIndexMapFrom2DLayer runs once after dwdLayerMetadata loads (junction + pipe 2D URLs) to populate maps for SceneLayer dynamic coloring without querying the scene layer directly.

7. Where maxRecordCount Comes From​

pageSize is not hardcoded in application config for network layers. It is read from the loaded FeatureLayer after the service metadata is available:

  1. layer.maxRecordCount
  2. layer.capabilities.query.maxRecordCount
  3. layer.layerInfo.maxRecordCount
  4. Fallback 2000 if none are set

Network hosted layers are created with only url: config.serviceUrl (see EsriNetworkMapPage.jsx); the SDK copies maxRecordCount from the feature service REST definition when the layer loads.

Publish-time source (monorepo): dwd-network-model-pipeline/arcgis_arcpy_utilities sets maxRecordCount on the service definition from max_features_per_request (default 1000 in arcgis_config.json). Older or manually published layers may differ.

Not the same as:

  • maxRecordCount: 32000 in shapefile upload publish parameters (EsriNetworkMapPage.jsx) — shapefile workflow only.
  • maxRecordCount on individual queries in EntitySearchModal.jsx — search UX only.

8. File Reference​

FileRole
src/map_utils/esri/applyV2SnapshotDynamicRenderers.jsqueryAllFeaturesAttributesOnly, dynamic apply + UniqueValue fallback
src/map_utils/esri/snapshotDynamicArcade.jsArcade Dictionary serialization, index field resolution, payload size limit
src/pages/EsriNetworkMapPage.jsxuseEffect → applySnapshotDynamicNodeLayer / applySnapshotDynamicLinkLayer
src/pages/Esri3DNetworkMapPage.jsxbuildIdToEngineIndexMapFrom2DLayer on metadata load
dwd-network-model-pipeline/arcgis_arcpy_utilities/src/arcgis_uploader.pyService maxRecordCount at publish time